About Apache Airflow
Apache Airflow is the conductor, not the orchestra — it doesn't move or transform a single byte of your data itself; it just makes sure every other system (Spark, dbt, S3, Snowflake, your own scripts) plays its part in the right order, at the right time, and tells you the moment something goes wrong.
What Is It, Really?
Airflow is an open-source platform for building, scheduling, and monitoring workflows — expressed as ordinary Python code rather than a drag-and-drop canvas or a wall of YAML. A workflow in Airflow is a DAG (Directed Acyclic Graph): a set of tasks with defined dependencies, and no cycles back on themselves.
It was built to solve one very specific, very common problem: data pipelines that used to be a folder of cron jobs and shell scripts, held together by hope, with no visibility into what ran, what failed, or why.
from airflow.sdk import dag, task
from datetime import datetime
@dag(schedule="@daily", start_date=datetime(2024, 1, 1), catchup=False)
def hello_airflow():
@task()
def extract():
return {"rows": 1200}
@task()
def report(data: dict):
print(f"Pipeline processed {data['rows']} rows today.")
report(extract())
hello_airflow()
That's a complete, real, runnable DAG — a handful of lines, and Airflow already gives you a UI to watch it run, retry logic if it fails, and a full history of every execution. Strip away the specifics and every DAG, no matter how complex, reduces to the same shape: a start, some work, an end.
Figure — the Graph View for the DAG above: three tasks, two links, all green because the run succeeded.
Zoom out from any single DAG and the Airflow UI also gives you a dashboard listing every pipeline you've registered, its schedule, and its recent history:
A Brief History
| When | What Happened |
|---|---|
| October 2014 | Created at Airbnb by Maxime Beauchemin, to manage Airbnb's own increasingly tangled web of internal data pipelines. |
| 2015 | Open-sourced. |
| March 2016 | Entered the Apache Software Foundation Incubator. |
| January 2019 | Graduated to a full Apache Top-Level Project. |
| December 2020 — Airflow 2.0 | The single biggest leap in the project's history: a rewritten, highly-available Scheduler, and the TaskFlow API (@dag / @task) — the Python-native way of writing DAGs used throughout this course. |
| 2021 – 2024 — Airflow 2.x | Steady maturity: Deferrable operators (waiting on external systems without blocking a worker slot), Datasets (data-aware scheduling instead of time-only), a redesigned Grid View, Dynamic Task Mapping. |
| 2025 — Airflow 3.0 | A second major leap: the workflow engine split from task execution (the Task SDK), multi-language task support, Datasets renamed to Assets, and DAG Versioning. |
Airflow is, today, one of the most widely adopted pieces of open-source data infrastructure in the world — the de facto default whenever a team asks "what should orchestrate this?"
This page is deliberately just the headline version. For the full version-by-version feature comparison and the complete ecosystem timeline, see Airflow Ecosystem & Version Evolution later in this module.
What Problem Category Does It Sit In?
Airflow is an orchestrator — a category distinct from (and often confused with) processing engines and streaming platforms:
| Category | Job | Examples | Airflow's Relationship |
|---|---|---|---|
| Orchestration | Decide when and in what order work happens | Airflow, Dagster, Prefect | This is Airflow. |
| Processing | Actually transform the data | Spark, dbt, Pandas | Airflow tells Spark to run — Spark does the work. |
| Streaming | Continuous, low-latency data movement | Kafka, Flink | Airflow schedules batches; it does not process streams itself. |
That single distinction — Airflow orchestrates, it does not compute — is the most important mental model to carry into every page that follows.